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To avoid duplication in such an^ exploratory development program, the approach was to tap the wealth- of 
existing, but scattered, soUrces.pf available hard data cojfceming job performance measurements, to structure these 
data as they applied to the mfcsurement of ability to perform electronic maintenance tasks, to analyze them in' 
relation to current Air Force practice and to make recommendations for the development and tryout of effective job 
performance measurements for Air Forc§ electronic maintenance. * . , 

PM)er and pencil testing procedures are used almost exclusively for determining which g^xsonnel are selected 
for training, for determining student progress while in training an^ for determining the promotion eligibility of 
personnel assigned .to field maintenance units. A number of studies are cited which indicate that 16w correlations 
were obtained by comparing job task performance tests to pap^r and pencfl theory tests and to job knowledge tests. 
Several studies also are cited which indicate , that the traditional theory content, found in most ^electronic 
maintenance training programs, does not contribute a great de^ to the ability to perform electronic maintenance 
tasks. A fun application of the modern technology for, technical training development would solve the course 
content problem. This technology requires a systems approach to training program development in which training 
objectives-are based on a complete job task identification and analysis. Criteripn referenced Job Task Performance 
Tests (JTPT) are required to determine if training objectives are achieved. Howe^r,.a serious gap remains in this 
technology^'since adequate guidance is not availabler for the development of JTPT./^ 

As a result of this analysis, it was recommended that c^prehensive exnforatory and^dvanced development 
efforts^conceming Air Force maintejiance should be establishe^d and funded. These programs should systematically 
and comprehensively identify and 'solve problems concerning maintenance practice in the field and problems 
concealing the selection and training of maintenance personnel. A necessary first e.frort should be t^ gather and 
study Hard data on how well maintenance men.can perfo'rm the key tasks of their jobs. Also, based on the results of 
this analy^ a contractual effort was initiated to dejklop criterion referenced JTPT for electronic maintenance tasks. 
This effort was followed by an attempt to develpp both graphic and video symbolic substitutes of improved 
empiQcal validity . The results of these efforts Ve reported in Volumes II, III, and IV of this series of documents 



ERIC 



. ^ UrjJ^lassifieil ^ 
\ r= ^ 

SECUFf|TYf%.A^IF1 CATION OF T HlS^P AGErW^«n Dmf Enffd) 



SUMMARY 



Problem 

The application of the technology of programmed instruction in Air Training Comniand resulted in a 
number of lists of so-called measurable behavioral objectives. After reviewing several of those lists in 1962, 
the writer felt that the- behaviors listed were verbal behaviors that could be conveniently measured by paper 
and pencil objective type tests, but that the listed behaviors were not necessarily the behaviors of the jobs 
for which the students were b^ing trainedJ^e felt th%l this was especially true for the key behaviors of 
maintenance jobs. Late in 1962, the author prepared a paper entitled Performance Testing: Testing for 
What is Real (Folej^, 1963). This was followed by, a Bibliography oh Maintenance Personnel Performance 
Measurement by Askren ^963) and work on a draft (not published) of an annotated bibliography. by 
PorterfieW in the same year.^All of these efforts indicated that there was probably a lack of job realism in' 
the formal measuring devices usyd to ascertaip the training success and promotion potential of maintenance 
personnel. They also indicated IJiat although a rather extensive technology had been developed for paper 
and pencil testing, no well-Structured technology or guidance existed for the development and- 
adminiitration*of job performance tests. These preliminary works and findings indicated a requirement for 
exploratory development concerning job perforjriance tests for maintenance. These works also, indicated 
that there had been more reports concerning performance measures for electronio^maintenanc^than for 
mechanical maintenance. This emphasis is reflected in the following approach. 

• * ' ■» 

' Approach • ' 

To avoid duplication in such an exploratory development program, the approach was to tap the 
wealth of existing, but scattered, sources of available hard data concerning jbb performajice'ineasurements, 
to structure these data as they applied to the measurement of ability to perform electronic maintenance 
tasks, to analyze them in relation to current Air Force practice and to make recomlmendations for tfie 
development and tryout of effective job performance measirfemen^s for Air Force electronic maintenance. 
The most productive source of information directly applicable to proficiency meastirements for electronic 
technology courses was tjte Defense Documentation ^Center. Approximat^y eighty reports were found 
concerning different aspects of proficiency 'measurement for electronic courses. Many of the same reports 
were also referenced in the Psychological Abstracts, Applicable material, as well as basic references, were 
obtained from several textbooks. Although the Educational Index was a valuable informatronal source for 
general vocational and industrial education, it contained ve^ little reference materials applicable to 
measurement in these areas. o ^ \ * * ' . ' 

The literature on vocational ec^ucatioii indicates that .cgnsiderable emphasis has been placed on the 

* development of objectives and mateftals for hi^ly job relevant courses. However, very little attention has 
been given io the validity of measurement procedures for ascertaining that these job objectives hava^been 
achieved. , , . ' j^' ' 

" *" f' 

^Results ^ • - V- ' 1» • 

Paper and pencfl testing procedures are used almost exclusi^jely for. determining which personnel are 
selected for training, for determining student prpgress'while in training and for determining the promotion 

* eligib'flity of personnel assigned to field maintenance units^ This ^alysis of reported re^arch iiwiicates that 
these measurement procedures do not insure that tfiese personnef can perform the ^lifctronic maintenance 
tasks generated-by the electronic hardware they a^e assigned to maintain. Most ^f me j^aper and pencil tests 
used in training and promotion concern theory, apd job Icno.wledgel Studies ar&^ited which fndicate that 
extremely low correlations were obtained by .comparing job task performance tests to pjper and pencil 
theory tests and to job knowledge tests. Several studies are also cited which indicate that the traditional 

. theory content,' found in most electronic maintenance training urograms, does' not contribute a great deal 
to the ability of an individual to perform electronic maintenance tasks. However, it is widely contended^ 
that a knowledge of theory is a necessary prerequisite to the successful perfon^ance of sucft tasks and this 
contention is deeply imbedded in the electronic maintenance culture. A full application of the modern 
tedinology for technical training develcTpment would solve the course content problem. This technology* 
requires a systems approach to training prograA^ development jn which traiiiing objectives are based on a* 



complete job task identification and analysis. Criterion referenced Job 'Rsk Performance Tests (JTPT) are. 
required to* determine if, training objectives are achieved. However, a serious gap remains in tliis technology, 
since adequate guidance' is not avaflable for th/ development of JTPT. When administered, JTPT have been 
expensive in terms of equipollent, tinm' and /dininistrative personnel. JTPT should be utilized in spite of 
their high cost if they are^ the only enipirica|((y valid tests available. If eny)irically valid symbolic substitute 
tests could be, produced, they certainly sho/ld be used. After a study of the symbolic substitute Multiple - 
Alternative Symbolic Troubleshooting Tes/(MAS'^)|and Tab Tests, it was hypothesized that the empirical 
vjJidity of symbolic subs.titute'of this tyWmight be improved by increasing their realism. 

Conclusions 

Based on the results of this analysis a contractual effort was initiatecfto develop criterion referenced 
JTPT for electronic maintenance. This Iffort was followed by an attempt to. develop both graphic and video 
symbolic-snbstitutes of inifprbved emiirric^J^validity. The results of these efforts are repprted in Volumes II, 
in, and IX of this s^lries of docurrwnts (s^.e Preface). The preparation of a guide for developing JTPT is 
recommended. Comprehensive expltf ratoryl and advanced development efforts should be established and 
adequately funded concerni^ig Air tforce maintenance. These programs should systematically identify and 
solve problems concerning maintenance practice in the field and problems concerning the selection and 
training of maintenance personnel/ A necessary first eff9rt shj((uld be to gather and study Hard data on how 
well maintenance meti can perforip the key t^sks of their jobs* 

^ 
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Preface 



This doiiment represents a portion of the exploratory development program of the 
• Advanced Sy^§ms Division, Air Force Human .Resources Laboratory, Wright-Patterson^ 
Air Force Base, Ohio. This document is the first of four volumes^ to be published' 
concerning the evaluatioa of maintenance performance. Volumes II, Illjand ly are; 

' l! Evaluating Maintenance Performance^ The Development and Tryout of 
Criterion Referenced Job Tgsk Performance Tests for Electronic Maintenance. 
AFMRL-TR-74-57(II),Part I,inpress. ' , ' 

2. Evaluating Maintenance Performance: Test Administrator's Manual and Test 
Subject's Instructions for Criterion Referenced Job Task Performance Tests for- 
Electronic Maintenance, AFHRL-TR.74-57(II>, Part II, in press. 

3. Evaluating Maintenance Performance: The Development of Graphic Symbolic 
'Siibstimtes for Criterion Referenced Job Task Performance Tests of Electronic 
Maintenance. AFHRt-TR-74-57(III), in press. ^ 

5 4. Evaluating Maintenance Perforrrtdnce: A Vfdeo Approach to Symbolic Testing 
of Electronic Maintenance tasks. AFHRL-TR-74-57(IV),in press. 

The preparation of thesa^volumes has-been documented under Task 171010, 
Evaluating the performance of Air Force Operators and Technicians of Project 1710, 
Training =for Advanced Air Force Systems. The effort represented by ihis volume was 
identified as work unit >71010O6, The auAhor was the task scientist; Dr: Ross L. Morgan 
was the project scientist* ' , • 

The duthor wishes to express his appreciation to Dr. WilliararB. Askreit, Advanced 
Systems Division, and Mr, James Porterfield, Aerospace Medicay Research Laboratories, 
for their origin^ biographical works which were used by the author for the original 
identification oif' sources for Jhe author's dissertation upon which this analysis is based. 
He also Wishes to express his appreciation for the guidance given by his dissertation 
committee: Dr. Dennis H. Price, chairman; Dr. Howard B. Lyman, and Dr. Jack Code - 
aD of- the University of Cincinnati. The final product was influenced by many discu^ons 
with 'Dr. Ross L. Morgan and Dr. Gordon A. Eckstrand of the Advanced Systems 
Division. 
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». EVTALUATlNGMAINTENANCEPERFORNfANCE: • V 

-AN ANALYSIS • - 

' , I. BACKGROUND . V . - ' " 

This report (AFHRL-TR-74-57 (I)) is the first of a serie^ of four volumes pertaining to the 
inp^urement of job proficiency. It presents the background and analyses that resulted in the funding©/ a 
series of exploratory development contracts starting in 1969. T^ese contractual efforts initially concerheti 
the development and tryout of criterion refejenced Job Task Performance Tests and later the de.velopment 
and tryout of both graphic and video symbolic substitute tests. The results of these follqw-on efforts are 
reported in Volumes II, III and IV of AFHRL-TR-74^57. Several in-house efforts and events prior to 1969 
were reflected in the writer*s preparation of the early work statements for these contractual efforts. 

Late in 1962, the writer prepared a paper entitled Performance Testing: Testing for What is Rear 
(Foley, 1963). the. motivation for, this paper jvas the review of several lists of measurable behavioral 
objectives which had been generated in Air Training' Command^ (ATC) as the 'basis for preparing 
progfammi^d instruction packages. The writer felt that the behaviors listed were verbal behaviors that could 
be conveniently measured by paper and pencil objective type tests, but that the listed behaviors were not 
necessarily the behaviors of the jobs for which the students were being trained. It jvas )he writer's 
contention that a large portion of the desired job task bfehaviojs involved complex per£;eptual'motor skills. 
He felt that the only way such behaviors could be adequately measured was by performance tests in which 
each t^st^ subject was required to^d^monstrate that hejcould in fact perform each key task of his job. As a 
.follow on to^this.ftrst memorandum report, early in 1963, Dr. William B. Askreh prepared 2i Bibliography 

^on Maintenance Personnel Performance Measurement X^^^^^y \943\ During the summer of 1*963, Mr. 

.James l^o^terfield, a graduate student from Kansas^State University, expanded and annotated the AsJcren 

bibliography. This draft bibliography was never published. 1 

\ ' ' ' ' • 

In 1964, Eckstrand publishexi a paper concerning the status of the technology of training in which he 

discussed measurement in* the context of a systems approach to training. He suggested the use of criterion 

referenced measures as appropriate for measuring student achiev^ent of carefully derived trailing 

objectives, stated in term* of what a graduate should do. He also discussed the difference between criterion 

referenced and norm referenced tests, and referenced Glaset and Klaus (1962) as his major source for this 

c^distincfion. . * ' ' 

In the 196S to 1967 time period, the writer prepared a review of the literature for his doctoral 
dissertation concerning measurement practices of Post-High Schbol Vocational Electronic Courses (Foley, 
1967a). These courses are ^ery similar in content to Xhp Air Force's electronic maintenance courses. This 
review of the literature .made use of the previous; biographical wofks of Askren and Porterfield as well as 
souroes discussed later. Thf writer's? thinking expressed in this dissertation was influenced by his many 
discussion with Dr. l(oss L. Morgan and Dr. Gordon A*. Eckstrand Many of the materials contained in this 
volume appear in the dissertation. The dissertation mate^als have* been modified to apply more directly to 
Air Force measurement problems. ^ . 

In 1966, the writer monitored and participated in a. field survey of electronic maintenance in the Air 
Force XFoDey & Elliott, 1967). The survey information gathered during this study indicated that 
maintenance practices in 4he Air Force coul4 be greatly improved. The authors were aware of the softness 
of data, gathered by interviews and observation, but as the^^Survey indicated, no recorded hard data existed 
concerning how well each maintenance man could perform his various job tasks. The generation of such 
performance data would require the adtiiinistration of objective jeb performance tests which were not 
available. . - * » • 

Further motivation an(f input for preparing the original purchase requests for the foDow-on 
performance measurement efforts (reported in Volumes II, HI and^ iV) were provided by two conferences 
on measurement which ^ere supported by the Advanced SystemriHvision, Air Forbe Human Resources 
Laboratory, WHght-Pattcrson Air Force Base>.Ohio. In March 1966, ttte writer.attended a conference on the 
Assessment of Complex Operator Perforrnance, This cooference was arranged and reported by Di. W. Dean 
Chfle>(l?6J), but was chaired by Dr. Arthur W."*Klehon of the University of Michigan. In October 1968, 
the writer participated in a confpnce chaired by Mr-, Melvin T. Snyder (Snyder, Kincaid, & Fotempa, 
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1969). This conference concerned human factJSrs testing. Both of these conferences indicated to the writer 
that there was an urgent requir^^nt for the development of more effective nieasuring devices for 
ascertaining how well Air Force pefedftnef could pe/form the maintenance job taskS generated by existing 
and future hardware systems. ^ ^ ^ 

- The purpose of ,the'wo|:k reported in this volume was 40 tap the wealth of existing, but scattered, 
scjurces of already *av^ilable information concerning job perforqj^mce measurements, to structure this* 
infontiation as it applied to %t measurement of ability to perform electronic* maintenaRce tasks, to analyze 
it ill relation to current Air Force practice and to make recommendations for the development and tryout 
^ of Effective job performance measurements for Air Force electlonic maintenance. One primary purpose of a 
project such as this is to eliminate as far as possible duplication oF work that has already been adequately 
accomplished. As mentioned previously, a major portion' X)f the information had been identified as part of 
the writer's doctoral dissertation (Foley, 1967a). But in that document the information had been analyzed 
and reported for a somewhat different purpose. . 

The most productive source of information for these efforts has been the Pefeose Documentation 
Center (DDC). Approximately eighty rejferts were found' concerning different aspects of proficiency 
ftieasurement for electrpnic courses. Many of the same reports were also referenced in the Psyfhohgical 
Abstracts. Applicable materials, as well as basic references, were also obtained from texts by Super and 
Crites (1962), Thorndike (1949), Travers (1950, 195*5), and Michaels and Karnes (1950). Although the 
Educational Index was a valuable informational source for general vocational and in'dustrial education, it 
contained very Uttle reference mat^ials applic^le to measureit^nt in these ^eas. The literature on 
vocational education indicates that considerable emphasis has been placed on the development of objectives 
and materials for "highly job-relevant courses.-^^llowever, very little attention has been given to the 
development of valid measuremenj procedures for ascertaining that these job objectives have be^n achieved. 

» n. THE CRITERION PROBLEM^ 

In most cases when we wish to determine whether an individual or'grdup of individuals can perform 
b^aviors in a given environment we capnot simulate all the desired conditions. If we can produce a 
measuring 4evice that actually measures the abflity to perform the desired hfehaviors under all the d^ired 
conditions, we, have an ultimate criterion measure. But the fact that we usually cannot develop such a 
devjce fonies'us to-^settle for a secondary criterion measure which is, ^"best, somewhat different,than the 
ultimate criterion measure. This dif^rence betweert the real world and the simulation of the real world for 
testing purposes is called the criterion problem. - / * 

A common ekample of such a criterion problem presents itself when we attempt to measure an 
individual's ability to drive automobiles. To measurt such ability cojnpletely we would have to devise a test 
•that would measure his ability to perform all driving tasks of airautomobfles, on all types of roads, in all 
traffic conditions, under aU types of weather conditions, whether he is being observed or not. It is obvious - 
that it Vould be virtually impossible to meet -all of these conditions under practical testing conditions. We 
therefore, settle for a less rigorous test criterion. We assume that he can drive any automobile adequately, if 
he demonstrates in a performance test that he can perform most driving tasks in one automobile, in normal 
traffic, while being observed. * . . _ ' ^ . 

• But many times, it is inconvenient and considered tol)^ costly to administer even such* a d^ivet 
^^^^ ^ ^"^'"P^ ^^^^ 'develop a paper.and pencil test whlqh wfll determine that an 
individual can drive adequately. But sUch a test cannot be considered to be valid substitute unless a hi^' 
empirical relationship to the criterion measure can be .demonstrated. In the practical world of test 
development the driver performance test^wuld be considered an adl^quate near ultimate criterion test for 
validation of such a paper and pencil, substitute. Many tiriies suclfk paper and peijcil test is used without 
being validated against such a near ultimate criterion test. The use of such an unvalidateS test is an- 
extremely dangerdus practice, since it is assumed%by most users that it measures an individual's ability 'to 
drive when m^fact we ate not sure what it is measuring. j 

. TOs criterion problem has long plagued measurement theorists and practitioners^ as weO as curriculum 
researchers. The use of job tasks, and performance examinations based on these tasks as near ultimate 
cnteria for ^valuation of selection, devices, was firstemphasized as a result of the work of Army and Navy 



Wrement psychologbts daring Wprid War 0. In 1946. Jenkto discussed the pAm in Kght of the 
' experiences ofNavypsychologbts in an article tiUed "Validity for What? ' 

PwchologisU in general tended to accept thettcit assumption that «?rileria«*te either pven of God or just to 

bTfound^ing about....-n.e novice of 19*0. searching through many textboola and 
" ' would have bin led to conclude that expediency' diaated the choi* of cntena and that the convenient 

availabflity of a criterion vvas more important that its adequa^f. ^ 
. Iii-1964 WaHacc presented a paper at tKe annual convention of the Am^n«n Psychob Association m 
w#i he indicated that much of what Jenkins said in 1946 was still true (Wallace. 196Saj. 
^ 'Supe?and Crites 1962). 'and Thomdike (1949) haw discussed this same problem in-light of their 
Amiy ej&erience. They have indicated that, although course success was a readUy milaWe v^^idation 
criterion -for prediction, success in the course often had little influence on job success. One of niomdike $ 
examplesfromthe Army Air Force (AAF) is as follows: ' , 

Early in WW n. gunnery schools in the AAF placed a good deal of emphasis in «hfif P»8«'" »" jf'fif.^ ' 
, n^rJfenclature of machfae guns arnl tunets and orf being able to express ^'^fjf^^ Soi bliS ' 
^ -equipment. Using grades obtained in this type of program as a cntenon. it was possibte Jo develop a battenr 

7f verbab' tests which.gave a substantial prediction of those grades. Actual combat gunnery, however. 

p«a no iecial vertel denunS. and STt^pe of mcmonter training referred to *<n«.w« event««By ^ 

done away in favor of more and moreUainin(jm the actual assepiWy inamtenance and finng of the guns. 

Tl^S natwe of grades in gunnery tel5&l chafed correspondingly. With this change m the criftr^on^ U« 

vaMities of verW. selection teslTUrgely disappeared. TTough there is no 
■ selection of g^rs in terms of verbal abilities would have done any actual harm to the .fi^ output. « 

«Sy woSdtave been wasteful and ineffective. Selection *ouW l^ve been based on the vrtlevant 

variance in the partial criterion of training grades (Thomdto. 1949. p. 13^^ 

Thorndike goes on to clarify the relationship among predictive tests. ^^aiSig and jo^ performstoce. 

Theie are many-other instances, both in and out of thetnilitary situation, which flCfl*mic«i»d« are u^« 
crhcrion m^OT* *ecau« they are conve.»iently/^ii]&W9 ar^l-because they appear to possess a rath« ^ 
saSoryTe^ of rdUbility Tliese grad^ cXneS&Vbe predicted with to success. 
TXxnmy U luted into a,««e of uitisfatt^w^ accompmmerU by thb "f^««/''.'^*f '^^f j*'"- ' 
' Sometimes, of course, his satisfaction may **Jtified, because the nature of the trammg is based on 
V^S the same attributes as perform«Ke^e job. The possibflity i*'*!*'^* 
tl« ikortton of academic'adilevement that we ^edict with our tests b not the part that is relevant to late 
succ« on the job. Whether the criterion is grades ot some other type of recorf. we must always exam^ , . 

criticaly tajudge whether the aspects of the criterion which we predict wiU be relevant for the ultunate goal 
• (Thorndike. 194^pp. 126-127). 

Since -an electronics training course has as its ultimat^objectlve job perfonnance, the ultimate 

criterion against which the course content and course measur^nt should be validated is the job for which 

the student is being prepare4. WaUace (196Sa) very succincUy expressed the criterion problem as it appUes. 

to electronics personnel. r - ; . * ' > * ) ' ' 

Alf of this is prelude to my mainthesis which is in no sense revolutionary, original, or coirtroversidji sUte it 

^ because it is honored in the breach. It is that th? nature of our proficiency measures determin^ how we 
select, classify, train, maintain and assess our human resources. If the measures are largely irrelev^t to the 
• jol» we wanVdone. we" wM select the wrong men. dassify thern Ijr. J»>d t«ta them 

true because these proficiency Jneasures are. or should.be. the crijeria agaimt which we validate «lwtion 
and classification procedures and -evaluate our training content and methodology or oUr V^V^^^ . 
techniques. IhUs. if I use a test of advanced electrpnics theory as the pWficiency measure for/eectroiUM 
maintenance and as the criterion agaimt which to evaluate a lest/for selecting men to go ^""^IT^^V^^ 
training. I wiU end up choosing a selection test which rejects m^ who are not weU f'»« 
reading and arithmetic abflity . In the prodfcss I might reject a great many who are outstanding in|their abmty 
to get their hands on a piece of machinery and make it worlc. I might also accept a number who (Uke myself) 
arft so UcWng in the rim^ejt manipulatWe abflity that their hands could have been cut off at ^he wristt at • 
birthwithouta«srio\)»ly affecting their outputs. So, when I dfccided what proficiency measures to use. I also < 

deddedwhatkindof men I washing to put into training for" the job. . j ( • 

But it doesn't end there. For when I now approach the problem of holJ to train men to perform the tasks 

inwlvcd In the job, I must makeltecisions about what should be taught and what methods should be us^ in 
' teaching it. The only way I have of reaching sudi dedsions {except by divination which is. admittwlly, not a 

rare pK)ccdure) is to mcasureiand compare the performances achicyed ^ith various cunicula and 
' methodoloaes.*So, iiS the caseW the electronics maintenance course. I put in lots of reading abou ^ 

eWctronics theory and Jvpjodua aaduatcs who can read andP write clccUonlcs theory whfle thdr equipment. 

deteriorates in fiopcless inopcmttvcless (Wallace, 1965b. p. i). 



Ftederiksenhas.expressed.a similar view.' ./ ' '■ ' ' 

h should bc,*obvious that one. cannot hoge to ptovidc pppropriate metKods^for evaluating instnictipnal 
-outcomes unless ht has a clear idea of what those qutcomes should be. Yet evaluations of 'instructional . 
programs are sometimes made in which criterion measures or^ choseh because of (heir easy avaflability rather 
^ than through a careful study of what the students are supposed to be able to do as a consequence of trainipg 

• . (Ftederiksen, I962,p. 324), . ' * ' ' v * . > ^ 

•TheTproblem is made more complex by t'he fact that a ^ven'.course often^is used to train many 
individuals; to Work in a, variety of jobs in the electronic field. To carry the logic of an ultimate criterion to 
its conclusion, each* of the; jobs^is an* ultimate critepioi\.sTp include- jof these in a course or its 
measurement program would be impossible. TTiis does nof nieari, however, that , we should ignore the 
prpblem of^preparing people for specific job. tasks that are common Or of meaguring thteir pr6fi%ncy in 
these tasks. We, can examine the electronic. jobs as they'exist tj>dayaand det^eonine jthpsetaycs wld^ a'rV 
found jij all or practically;all electronic jobs. We can even predict with-soifie degree qCcertainty the tasks of 
-futurejobs. - * t. - 'K ' -^•^i''- ' ^ V':^- a*;' 

Thorndike (1949, p. 121)-dijfe'rentiat^ three categories cjf criteria: ultiinate; iMermed^tei- and 
Immediate. He indicates that **A...re'ally complete ultimate criterion.'is multiple land compjex in almost^'". 
every-case.'* He also* suggests that it is possjbte^otegree oi> certain final job behaviors as being the closest 
approximation to the ultimate jg«>al. prederiksen makes the foUowiftgrappJicable .statement:: 

' ' \ \ ' " * ' - " ' * ' \ 

The objective^ presumably. Is to get as close as \i feasible ^ t^e ultimate criterion; but as has just beens^en, > 
whe^ one gels tqo close to the real-life situation, control of the conctttidns for adequate observation is lost. 
. 45 Observations of realjUfe behavior is ordinarily nbt a suitable technique'for measurement: The type otmeasure 
1 that is recorftmended for first^ consideration Jn a training evaluation -study is the type which most closely 
K Approximates the reaHife^ituation, that whii^, m this chapter, has been called eHcitinglifelike.behaVior. If it 
is not feasible to wait for the behavior to happen in real life, then lifelike occasions can be provided for the 
beh^vicr to occu/ina test situatiort(FrederikseA; f96z,p. 334)* ** - . i . - 

Admittedly, W examination made up of tasks removed from their actual job enyironfhent-.is jiot an . 
ultimate' criterion, test. Uilder actual job situations, the graduate may have to perform these "tasks in 
(Stamped quarters; under stresses 6t time, noise, heat, or cold', or with an excited boss interfering^ Theie 
conditions Of ^.tfe^^are- usuall/' not constant variables, but change from day to day and ftom hour to hour. 
, The asBumptit5n»usually has to be made, that the individual can perform a taslc under conditions^of strtss, 
jpfovided he*cah perform the same tisk well under jiormal ponditions. A formal performance fexaminadon 
^has its own set^of stresses, which njay not.be the SSame** job Stresses, but their presence may tehd to-offset 
the lack of job srtresses. 'tormal, job task performance, ej^inations are the closest usable sunulation of the 
real maintenance jpbs presenfly available. They are far bet|w than no performance tests a]^all.< 

• -•. ■■ ■• - . f ■• •" 

' * *t " ^ ra. WHAT JOB ACnVITIES SHOULD BE MEASURED v * 

^ * " ; • ' . • ' : ' ' ' ' 

One of thfc problems which has confronted the measurement psychologist when-heTias tried to . 
develop job task perfprmance tests for electronics maintenance has been a lack of a usable classification of, 
job activities which would be definitive enough to develop diagnostic test procedures!. The problem has 
been to fijidcategoK^s of behavior which are. meamngful to training'and maintenance personnel and which 
lump simil^ or clos^y relat£d behaviors into each category. In 1962, Bryan summarized^and classified such 
■ activities. In 1966, Folley discussed ^e job tasks of the electronic technician based pn many sourcesm m 
unpublished paper. From, the Folley discussion j the.Writer developed the following* list of tasks or activities: 

1. Job activities or tasks associated witl\ electronic equipment: ^ 

\ a." Performing* equipment checkout procedures. ' ' . ' • , 

'b. Adjustin^aligning, and calibrating. . • - * ^ ^ 

/ c. Replacifig of components — removing and replacing. 

• d.. ^Isolating between««tage faults to particular state (or. functional unit ^ or physically^* 

• • . " • .: repfeceable unit). ^ * . ; > \ • . > 

e. ' ^ Isolating witMn^stage faulis to 'defective component '(tube, solid state* device, coil, 
c^acitor, resistor, etc.). • - • * ^ 



■J 



2. Information gathering activities of electronic techmcian jobs . ' ^ . 

a/ UsingVosciHoscdpe to: (1) measure i'oltag^, (2) measure frequency, (3) compare 
waveshape to waveshape standard, and (4) m'ake high accuracy time base measurements 
* and comparisons. : c > ; j 

b. Using the electronic voltmeter to measure various ranges of v^tagps in electronic; 
- equipment. ^ * . 

c. ' Using the (Ammeter* to measure direct-current resistance in electronic equipment. 

d. Using the signal -generator to inject standard or know signals into equipment for test 
purposes. . , ^ \ , ^ 

e. Using the tube checker to estimate quality of electro^ tubes. ' ^ 

f. Using the transistor checker to estimate quality of traiisistctrs. 
Handtcfei activities of electronic technician jobs: .'. , , . 

a. • Using screwdrivers. ' ' . " ' 

b. Using plfers. . , ,r - * 

c. . Using diagonal cutters-. 



3. 



d. 



e. 
f. 

g- 
h. 



1 



Using soldering iron. 
Using soldering gun. . ...t^ 
Using wire strippers. 
Using machinist's wrenche^\. ♦ 
. Using light^chfiiik^^^ , ..^ . 

This list is not offwwFWW^-i^^^ that can be considered common 

. _r ;^u- Yi^^^^'ji 2*^*- s«.^ui^«f »yiJ^J;«r.^va o/**j^^*s*»rt*t!ttiiTft^«torihh'erat'tn actual maintenance 
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to most of the jobs. It^4iof^cllide aditrffl^fr^ve acti\^tierthar;arH^hera»:t9 actual maintenance - 
actSritlfes. Tfie five items in paragraph 1 of .thiSxlassific^tion Qoncem taScs pi^iyiti^s associated with . 
many actual items of equipment With \yhicji the technicians work,^ The six it^ras W Paraph 2 are 
informatio« ^thering activities which allielectronic technicians will encounter in the perfdm^.e-'ofihe ^ 
five ^quipirtent activities. From a logical point of view, it would appear thaj any training program ;fhat~^-;iv 
\as Its objective the preparatiph of students to work with electronic hardware, should include a Urge amount 
lof^p^ractice in the use of such information gathering devices. Paragraph 3 includes most of the small tooB 
used^by electronic technicians, t - r . 

This tlas^fication represents" a'^onvenient structure for the development of Job^Task Performance , 
test (JTPT) procedures and^apcompanytng scoring,schemes which have, broad apphcability^ to specific . 
electronicsjobsand related training. . ^ • 



/ 



' TV. CONTENT VALIDITY VS EMPIRICAL VAUDITY 



The maihtenaiice of 'D^fense^ardware syst^msnis an absolutely nece^jy .acUyity, but' its cost is/ 
extremely high. Is thU maintenance, cost too high? No-onejealTy Ichows the~answer^ And impqrtajrt ; 
unknowns include the ladk of extwisive hard data on the efficiency and^skiU oOhejQwintenanc^ 
and the lack of extensive hard -data on the effectiveness of maintenance training. By hard data~arr niean^t 
empirically valid data. The limited amoujij of published data available Jlndipatesf' tltat, m^nr*!"'*™'^ 
technicians are not adequately trained to perform th> Usks of their assigneQ jobs and thatlthey doii^^y-. 
perform many of their job tasks with a hi^ degree of proCiciency (Andefsoii, 1962a; Fe^yA Elliot^.- 
1967) Yet each Air Force electronic technician has a record of passing school marks and of a passiftg sqore / v. f0Qi 
on thd^ Specialty Knowledge Tests (SKT). But there are no hard data available on how weU he can jpe^Ktnry. -j- ri; 
' the actual tasks of his job. How valid then are the school marks and thc SKT scores? It depends on ' X| 

type of validity we are talking aboOt.^Tie discussion that follows considers Jhree types of validltyi-fact . 
validity, content validity and empirical validity. . 
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Face Validity means that'4he test looks as if it should be yaJS (Lyman, 197!)/ ^ /' ' 

* .■ Content Validity is somewhat simlliar to face validity, but i^-more system^c^ai^ sophisticated. _ 
^ naEn(es for it include logical validity, course validity, curriculaj.validity.arid texfboiik/<^alidi"ty/feth face anB 
. .content validity are non^^fistical (Lyman, 1971): Thi^ dej^ijition of poriteQt xili^ify is aBp^edio cT)uise , 
content. Tests can be constructed to have job;conlent v^ldity. This subject discussed ^ To obtain '/ 
'good content , validity for {course testing some so/t'of test Vblue'pri^f ' i$ u«uaHy»usfed to insure that all > 
: asjiectsoftheiubjectmatieror ofthejob.a^^ • - '^•Vf ' ' Y ^ .'".-'^ . ; 

. ? The tests given by teachers or professors, in acaidemi^ education situ^ti^^^ validity. 
' If iKey don't, the students cpmplain. If carjjfuUy prepared, some of tha<e tSts^'have cbiitent (er'ccmree) ' 

validity. Many of th- — w.k,.it, L . > , 

' have good content 
be'said for the 

Empiricai Validity is cri^erion-referencejdf validity and "is impl|cd;^(ji4^ is used tp 

modify validify. TTiis sort of validity is mo^ important/In pw/fctt/ «i(qa^ifs. }{ovf well d^o^s, the te'st 
measure what we x:^^iwi^^i Mf^u^/*^. .J^L^ *i. i 2- ji ji^l- -r^'i^i.^ ' " ' 

, tq some criterion 

^ is norf-statistic^; ki others, it must be defermjiied by statistical^^ 

? : ^ 4f a.testcojiild be constructed whjclf was an ultimate cnj^rion incasuie^for k maintenance 
Job, it would have perfect-empirical validity. It Would, 2i$p^^i^V^'.^^ it. ^ 

have to contain all of the job tasks performed in a^ritel^'^^oniy^^ AijRo;ug^ the' tasks that, aii . 
electronic' ipaintenancj? techmcian must perform can',teld^'ti0'ecti:fe3ler^i^^ in. 
whfch each task is p^rforme^'vai|^'fr6m perfonnaQce'to-pcltenr^ fof'is a t^t . . 

environment which closely appiroxiiriatw a typical i^-Ufe-sjtua(^OI^l^ test/wWch/is a . 

;.P^rfw^ 'ultimate criterion. measure, the^ & no test that has pe^>^pig(^;^^^ IfJglJttasks'aie-J 
carefully and systematically sdected for a tesi bat^eiy) so as to be^thlQ^^^^ ofihc 
job, v»d the test subjccts^are requircd^o perform the selected tasks in irj^eiivitl^ 
* job environment, iij:on logicaUy/be a5mrke4j&^ we^^^^sxlpse as fe^l^^lth^f^dtliTUittf^^^^ (The 
, performance of eidi selected task is afiviniM It cBiik^^'i^fne^ fi^t a bafteijt . 

made of such ^^sts'has j)ra6tica) empfl^c^vvalidity and'j^t(^con|pnt vali^»:(iS^iadn'^M§bse for 
presenting the/cl/i^ificatipn, of maiirf^n^ was ta .pioyide a frame of 

reference syiteri^ljc siie^ of neac^ultlmate criterion 

^ te§ts.) Th^^miS^ical vaadlty:^^ obtained hy the maonerjii 

. whfcl^ thi batteiy has p^n c^T^t^^e4[S^ non^tis;tic$a type* ofValicliV Such'a battery of teSs 
woiild bfeyob critejip^ 'f^feijM^^^^^ m^VMayfiis such tests arc called ^criterion referenced Job Task^ 
' ^ Petfornrai«:fifr«its^'''^ .vT^v, v . . . .. . ^ ^ 
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each test subject's ability to perform dairi^ey'^asktcf^the jc^^ for which he is being trained or tjie job to 
vrfiicTi he is assigned." The sections that fdlew present what has been found in the research Ulerature • 
concerning the empirical validity, of coUrs».'cO|^e'nt arid empirical vali4ity of the various typ^|)f tests 
now utflized, not owy in maiiiitenarice -trainifig ptogtains, bjitialso Jn field inaintenance units. .. ^ ' V 

" - . ^ V/THEViaibrrY^FiofeTA^^^^ I 



' Since p^er and pe^h'^tests ahd/rther^ufestitutes are mUcJiWd^^^^^ to admir^terand le||time 
consuming, they .are used.extensivel/ as measures.'bf acjh^vement-. .Examples of other substitute m^u^es 
include syifibolic equipment test<sucK.as. the tab Xe^ (Crpwder, Morrison, & Demarfee, 1954) aita the 
Mqltipre^Alteri^aUve.Symbolic Jroubleshoi^ Rigney, Bond, & Summers, 195^ as 

wdl a^. peet - ratings and^ supervisori'' ratings: Mravers <19SS) ^escribes various substitutes fo^ajob 
p^formance t^sts sucti'' asloral questioning, printed verbal tests (paper and pencil)^ and illustiatedv^or 
pictorial tests. / \ .* . /. ' . ' j , V}^, 

Research evidence, however, ^ves/a rathejr low xating to all of these substitutes. Table 1 shoys 
conelations thSf have been obtained by cpmparijig Job T^iiji Pejformance Tests (JTPT) to theory tests, ajrf 
to job knowkjdge tests.-lEhe lattef two Wpaper aiid pe'he'il iests. Table 1 also includes correlations wifli 
school marks, in most c^W', schoolmarks have been heavilyv weighted with thepaper and pencil test scoresl 
.An examimtion>f.d^^ correjatibiis. of JTPT scores with theory test scbr.es are 

gen'er^ify^j^nfey^ • ' 

probably tht mo^St^te^mye^sMidy regarding sjibstituttens of a paper and pencil achievement test for 
actu^-^J^E?^^^^ Hunian Resources Research Office Projejct REPAIR (Brown, ZayAors, 

Berhk^in, & Shqen^^ tp ch^ck the "effectiveness of the revised repair course against ihc 

tra<iitiQnal ton^^ a ^proficiency .battery made ip of six separate tests was devised, ITie battery included 
fo6r JTPT/JT^'>v^re 'developed lah admiiustered in the areas of troubleshooting, ttfst 

equrpmentj^r^pdfc'skillir, and alipment. Two paper and pencil group' tests were given on the use of 




^liiis' t^5t /con^ted ' of seventy-seven multiple . . _ - 

tioiitijeiiootihg, use of schematips, use of lest equipment, operation checks, and alignment. As presented in 
""Table,!-, the cprrefation bet weea this. achievement test and the performance troubleshboting test was .40; 
/withSlignmeiltW, .28; the repairs skillsi test, .19; with the test equipment, .29; with the paper and pencil 
, (fianuals test,' .5t; and with the paper and pencil schematic test, .51. The authors of the report state that,as 

.w.ould be ejqjected^ the correlations are the Hi^est with other paper and pencil tests; namely, the manuals 
/tjpsls^and sch'^matic tests. The authors also state that although the achievement tests correlate positively 

w^ltfi.mostQfdi|^ other tests in the battery, the correlations are not hi^ enough to justify the use of the 

achievement test as,a substitute for. the battery. of JTPT. This statement al^o would apply to all other tests 
*'preMntedih:f?ible 1^^ ,^ ^ * * v/ 

The Sai^^ (1955) sindy^ indicated somewhat higher correlations than the o^her/studlei presented'. The 
/ conelatipn between a troubkshooting performance test and a job knowledge test Was 55 and with school 
' m^.w'as jfi; Hpweve'r,.the troubleshooting prbjplems were performed on a very simple radio receiver built 
; specifkaUy for the study^ oii an W'n. chassis. Hie test subjects also used two very simple test equipments - 
;a vo]tptometet and^i^^'^^ All of the other studies used actual military electronic hardware of 

* ^^atiripmpi^iqty requ^^ ' 

,''.'''A>inpre>ecent study by Finuc^nc^ 0966) in the field of en^ne^pait in the Arniy results in very 
siinilar cprKJi.tisions. The resiilts of a JTPT on yarlpu^jgi^cts of fngine njaintenance were compared- with 
jpb taioyiietige tests .cpncejrr^d.j^th the same englnesflne correlation between the two types of tests was 

* jsirVfiich is; within the sam'e rai^e as the findings relative to the electronic maintenance field presented 

* '^ ..;,VSince, troubleshooting: ta*^^ responsibility of the maintenance 
tjx%rd^.2m, a great amount of the Research andydeyelopment effort concerting electronic-training has feeeri 
devptedno these' tasks'^-Si'muiators, .such as the tab Test (Crowder et al., i9^4\ the MASTS test (Grii^s ct 



Table L Correlations Between Job-Task Performance T^ts and f heoiy 
Tests, Job Kno wfedge Tests , and School Maries . 



Researchers 



Type of Job Task " Theory Job Knowl- 

Performai^ce T;est (JTPT) * Tests edge Tests 



ScMol 
M^rks 



Anderson (1962a) 
Evans and Smith (1553) 
Mackie et al., (1953) 
Saupe(1955) 
Brown etal^ (1959) . 



Williams and Whitmore 
(1957) 



Crowder et al., (1954) 



Test Equipment JTPT 
Troubleshooting JTPT , 
Troubleshooting JTPT 
Troubleshooting JTPT^ 
' Troubleshootmg JTPT 
Test;Eciuipment JTPT 
Alignment JTPT 
Repair Skills JTPT 

Troubleshooting JTPT 
(Inexperienced Subjects) 
(Experienced Subjects) 

Adjustment JTPT . ' 
inexperienced Subjects) 
j^Experifenced Subjects) 

^^quisition Radar JTPT ' 
rtlnexperienccd'Subjects) 
(Experiended Subjects) 

Target Tracking Radar JTPT 
* (Inexperienced Subjects) 
. (Experienced Subjects) 

Missile Tracking Radar JTPT; 
(Inexperienced Subjects) 
(Experienced Subjects) ' 

Computer JTPT 

(Inexperienced Subjects) 
. (Experienced Subjects) 

^ Total JTPT 
.^Inexperienced Subjects) 
(Experienced Subjects) 

Troubleshooting JTPT T 




,r .24 & .36 
V .38 ' 



.40 

9 



.23 
.15 

.02 
.21 

.03 
.14 

.24 
.20 

.09 
.19 

.08 
.06 

.14, 
.20 

M 



.12&.10 



.55 



.29: 
.28 
J9 



36 
22 

33 
3& 

.15 

32 . 

.24 
.14 



.18-32 



.18-.3.3 

;35 
(39 
.56 
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al, 1953), and the simaar AUTOMASTS test (Biyan et al., 1959) have beea developed. (The AUTOMASTS 
test wa5,essentiaDy a refined and automaticaUy-recordiilg form of the MASTS test*) The Tab Test displayed 
a schematic of the e^quipment. The subject lifted a tab^t each test goint to obtain information normally 
obtained by using test equipment on me real equipment. Hie MASTS test used corks instead of tab54.Two 
studies were made concerning the effectiveness of the Tab Test. The mo'^t extensive stydy by Crowder et 
al., (1954) found corrdations.of .12 and .16 for two forms pf the Tab T^st with JTPT Evans and Smith 
(1953) found a correlation of .07 using similar tests. Ihe MASTS and AUTOMAST tests were not 
compared with .real equipment JTPT tests. Several more recent studies have indicated that the electronic 
technicians twted could^. not\6btain information from their test equipments accurately (Anderson, 1962a; 
Poley, ISjitoPerhaps, the tarious electronic technicians used as subjects for the above correlation studies 
had the^^Bpfeakness. Tli«w . • j . 

Peef' ratings and supervisors* ratings also are puor indices of technicians' capabilities to perform job 
tasks. Crowder et al., (1954) found that results of jRpT correlated only .16 with supervisors' ratings and 
.23 with peer ratings. Anderson (1962a) .found, various correlations from '.19 to .41 between JtPT and 
. supervisors* ratings. Wilson has indicated that: * , * - " • < 

High intercorrelation of rating scales despite great ind extensive efforts to define rating variables in mutually 
exclusive manners have rendered ratings disappointing to anyone doing research in the field of selection and 
have almost eliminated th'eir value in tijuning stQdies (Wilson, 1962, pp. 370-371). 

Thomdike gives an example in which airplane commanders were rated while going through operatfonal 
combat training. A rating on "likeableness" correlated highest of any of the ten traits rated with overall 
rating of suitability for combat flying (Thqrjjidike, 1947). The Fmucane (1966) study cited earlier indicated 
an insignificant correlation of .J4 between|*^TPT pf«engine maintenance jobs and the commander's 
evaluation report, which was a rating s^calff.vRating scalesr based on supiptvisors' opinion of how their 
technicians perform ftave not been very successful as measuring devices.. j > ♦ 

This writer's opinion is that much more, work can be done ta-jmgrQ^'symbolic maintenance tests as 
substitutes for JTPT. In the opinion of fh^ writer, higher correlationsdCoSld" possibly be J^tained by a 
different approach to the development of symbolic substitute tests. study of the TabdTests* report, of 
Crowder et al., (1954) indicates that the JTPT used as the*criterion Assure contained ^r^y idisUactions 
an4 interruptions to the subject's troubreshooting strategyXcognitive process) such as usiAg test equipment 
to obtain test point information. la addition to such interrtiptions to the cognitive process, the subject can 
obtain faulty test point information by improper use of his test equipment. In the symbolic substitute Tab 
Tests all of theM potential pitfalls' of the actual task were avoided. The subject' was given a printed test 
point readout. The iigection of equivafent pitfaDs into symbolic substitutes would possibly increase their 
Empirical Validity. ; * . ' * ? ""^'^ i 

Such tests, however, in their ament stage of development, cannot be usedits substitutes for actual 
JTFT with great (Confidence. Althoi^|ob knowledge 'paper and pencfl tests abpyt,th^ various job tasks 
appear to be somewhat bettei thanfkbstrkct^theory t^t5 for measuring subjects\ability to perform job 
' tasks, tl|eir correlatipns with criterion J'tIT*, h^ive» not been very high. The measuring devices that are being, 
used as substitutes. are certainly not good eaotigh to replace JTPT in electronic training c^nrsc^ and in fij^4. 
maintenance units.' * ^ • • * * 

i vi. TBCTO^OLOQY OF TRAINING DEVELOPMENT AND THE ' . : 

, ELHrmO^IC MAINTENANCE PkoqiUMS 

" Modem technology for devel(4i)ment .of training program? is based on a systems, approach (Eckstrand, 
1964). The application of the systems approach as described by Eckstrand would result in a job oriented 
training prograro that would produce men capable of performing effectively the ,^asks of the jobs for which 
they are being trained* * ♦ ^ 

:A key element of the technology i^ the care used in deriving training objectives based on a systematic 
identificatipn of the tasks of the job or jobs for which a training program is^ng developed, followed by a 
sy^titnatic analysis of the identtfijed tasks. The second key element is as important as the first. At the time 
the objectives are being formulated, i^the cnterioii measures are also being developed. Based on the analysis 



of each identified task, the content of the training program is developed. The criterion measures are based 
on the obfectives of the training program not on its content v o 

After students complete the training program, they^must pass the criterion measures. A student 
shouldnot graduate imtQ he achieves the criterion. \i a large number of students fail the priterion tests, the 
contOTF of the training should be modified until a major part of the student population cdn achieve the 

- criteffion. In the case of the training of electronio'technicians, a job task identification would result in some 
mix of the tasks mentioned previously as they apply to the specific hardware to be maintained. ^The 
objectives of electronic maintenance training programs, which are developed using this systems. approach*, 
should be the training of personnel to perform the tasks of their maintenance jobs^ffeclivdy. A^ordingly, 

« the criterion measures should be designed to ascert^ that fhe graduates can in fact perform these job 
tasks. The name now generally applied to such criterion measures^ which prescri6e a **go, no-go'* 
performance standard is criterion referenced measureS (GlaSer & Klaus, 1962). 

The development of the core of current electronic maintenance training programs predates by many 
years th^ technology^of training described above. Their current patterrt of content is a product -of the ^Vorld 
War II maintenance training efforts (Foley, 1967b) although some of the content pr^ates Worid War IL All 
of these courses contained the study of electronic theory followed by the study of equipment tb be 
maintained. The equipment portion of such courses in all cases'Jricluded a §tudy of the theory of operation 
including circuit analysis. The equipnient portion usually included 'ihands on" training, such'as checkO'utr^ 
alignment and troubleshooting: The amount of such •!hands on" actiWty varied from training center to 
training denter and at first even from instructor to instructor. * ,1 ^ 

The source of electronic theor^and ecjuipment theory of operat* n portion of these cou{^s-was t^le 
electronic engineering design community. These materials were academic in ftature and traditional academic 
testing procedures were used to measure them - mostly of the paper and pencil, multiple-choice objective' 
type. The "hands on", training content was not determined in'any systematic way as would libw be requTraJ 
by the systems approach to traimng development. It wias usually determined by^.what. ir^tructors of*a 
training program decided it should be .--The measurement of each "student's success was forced into the 
gra^lingsystem-used in the, ?cademfc portion of the course. ^ ^ \ ' 

'^^DuriM World War II, checkrooms, were established in technical schpols of , the Army Air Force, ^th 
written antf^practical (JTPT) were Carefully (Jeveloj^ aM administered. In most cases' the proficiency 
measures used were of th8 norm referenced type (Glaser & Klaus, 1962; Eckstrand, 1964). This pattern of 
training and measurement continued with i^npro^Jferpents in job relevance until Ihe mid-1950s. ^ 

In the. late 1950s, many checkrooms were eliminated and the "hands ,on" equipment "(S'ining was 
greatly reduced because of the expense. The result, was modification of equipment training to the 
orientation, "hands off type. The measurement of the. success in such training could only be of the paper 
and pencil variety. At this time^he electronic^ training, progi^ lost practically all of their job orientation 
and the use, of JTPT almost disappeared. The learning to ^erfbrm actual job tasks was delayed until a 
^graduate was assigned to a fidd maintenance unit. • ^ * < . ^ ... 

The theory content of the training programs was retained with little change and each stu.dent's 
success, in learning theory continued to 1:1c measured primarily by norm referenced paper and pencif tests 
wftfdi are the appropriate instruments for measuring this type of material The question is how much does 
such learning contribute to a technician's ability to.perform the tasks of his jol). And such relevance should 
not be assumed as true without proof (Formal mechanical maintenance training has been more job 
oriented than electronic maintenance- training. The .mechanical training retained its job orientation even 
when the use of JTPT was de-emphasized.) ' " > k * 

^ Although action wasjaken beginning in 1968 to increa^the use of JfTPT in conjunction with 
equ^ment training, the electronic training programs haye retained their great theory orientation. All of a 
student's scores from JTPT and from paper and pencil theory and equipment knowledge tests are combined 
into a sin^e course,score. Most of the,.tests from which the student scores are obtaiiied are based on the 
content of the training program and fiat from task content of the Jobs to which graduates are'Sssigned. As a 

- result, these training programs do not truly reflect the modem technolo^ for course development,- 
described previously, nor do. they reflect the recommended procedures for measuring course objectives by 
criterion referenced tests. /* " , . . 



*V1I. WHY FORNf AL JOB TASK PERFORMANCE TESTS ARE NOT USED MORE EXTENSIVELY 

Even though paper and pencfl tests in their cunent state of development have been shown to be very 
poor substitutes fpr actual JTPT, the use of paper and pencil tests still persists. Hie question, therefore, is 
raised as to WhV JTPT are not used. Harris and^ackie (1962) report a study concerning factors influencing 
the use^of JTPTin the Navy. They have summarized the reason^for not usii^gjsuch tests as follows: 

The dficrepancy between the generally fa^vorable attitudes of both sup^rvisorsjand instructors toward the 
^ concept of practical performance tests and the most limited use being made of them was a result of problems 

connected with their (development and implementation. Their infeaability was the primary reason mentioned 
" hi supervisors and Instructors for not using performance' tests. A related reason - tests had not b^en 

developed o^ were not available 5- was mentioned frequently by a supendsor^bout operating activitfes. 

The most prominent barrier, in the minds of supervisors and instructors, to the use of practical performance 
^ tests was the degree to whiph the time necessary for testing must be taken from that required for carrying out 
Jhe primary mfesion of the activity or school. Performance tests were gener^ly felt to require too mudi 
cquipjnent time and too much personnel time to be feasible (Harris &»Mackie, 1962, pp. 4-5). 

,Often the amount of ctime required to administer a good performance test is rather great. The performance 
tests used in the study by Williams and Whitniore (1959), required approximately ei^t hours to administer. 
The four performance tests used in project JIEP^R required a total of approximately ten hours (Brown et 
al., 1959). But in view of the relevance of JTPT to the job objectives of electronic training courses, the)low 
validity of symbolic substitutes for these tests, and the high cost of electronics technical training, 'the . 
rationale of I'too little tinie avaSable" is very weak. ' ^ ' " 

XnotheV.serious impediment to the use of JTPT is tfie lack of information on how to construct ^ood 
JTPT. There are maiiy textbooks on written tests. The onl^> text material of any magnitude that the writer 
has been able to find was, a chapter in Micheels.and Karnes (1950).JThe|ocational education literature 
indicates thaf considerable emphasis has been placed on the development oT job oriented course objecdves 
and* job relevant curricula* materials. But very little emphasis or attention has been^ven to appropiatc 
measurement procedures for ascertaining that these job objectives have been achieved. InsteaSof 
developing their own measurement; literature, vocational educators have .attempted to follovgfthc 
measureinent practices that have beeif developed by and for theacftemic community. This is a very s*ous 
weakness in the field? of vocational education sinc^ whatsis formally tested is usually tau^t better and 
learned better than those curriculum items which-are not tested (See* Figure 1). , . 

The Armament .Systems Peraonnel Research Laboratory (ASPRL), at Lowry Air Force Base, 
Colorado, a laboratory'of the Air Force Personneland Training* Research Center (AFPFRC), supported a 
rather^comprehensive research and development effort concerning the measurement of job performance. 
One of the results^oft their work was /I Guide for Use^in Performance testing in Air Force Technical 
Schools (Highland,, 1955). But it was published at about the same time that checkrooms were being 
abolished and perfprmanfce measurement was being de-e^nphasizcd in Air Training Command. Although its 
use in its present ^rm is^^iot reccftnmended, there is certainly a need for-such a guide on how todevelop 
and use criterion jrefereifced JTPT. i^s noted by Eckstrand (1964), >^'is a soft area in- thi^ modem 
technology of traiAng. It is too much* to expect technical training personnrf^o^repare good JTPT without 
adequate ih-depth guidance. (AFPTRC^and' ASPRL were abolished in)^1958>ftfc4Uesu)t, this important 
measurement work was discontinued. For a ^hort summary of the work of AFPTRC see G^aser^ 1 944,) 

VIII. THE JOB VALIDITY OF THEORY IN ELECTRONIC ^ 
TECHNiaAN COURSES * rV 

It is often contended that i knowledge of theory is an essential part of an electronics technician's 
training since such knbwledge.^ves the technicians the broad background necessary to perfprm many types 
of maintenance acthdties. The published literature presents no hard , data to support this widdy held 
contention. HowevSir, there are several published studies which support the hypothesis tha\ a knowledge of 
theory aS' measured by paper and pencfl te^ts does not insure the abflity to perform ijiaintenance tasks. In 
fact, ^here is some convinciiig evidence that there is little relationship between true ability to perform job 
tasks and knowledge of theory .as.measurefl-by paper and pencil tests. r \ . --'^ 



Lyman {1971) makes the extinction between qiaximura perfora)ance tests and typicbl performance 
tests. Performance tests as used \fi L7 man's discussion have r§ference to all types of perforcnance whetW 
they be verbal, manipulative; or c^inp lex. Since the word "performance" is used in conjunctiorv^Wlthrjob 
task performance tests, the term""effart" is used ,here jn order to avoid 'confusion. Whenp formal 
examination is given by a teacher, each-student usually makes a rather intense effort to prepare* for the 
examination and also makes a maximum effort during the examination. Micheels an^arnes (1950) fnahe 
interesting comments about this effect of tests on Jearnfng. , ^ ^ ^ ' 

Tests Provide an Incentive for Application: In order to^galn an understanding of ther interest of students in 
test results, one has only to observe their reactipns wh^ te$t papers are retumefd for review and discussion. If 
nhere is any question about this point, consider your own experiendissj when you were on the receiving end of 
Ihe tests. Think tjf the times you have put forth extra effort in getting ready for a test* Sdme readers will still 
be very much concerned with their ability to pass th^ tests that are constructed 'by their Instructors. They 
will agree with thfe authors that knowledge of a'forthcoming'test is a powerful motive to start studying, 

. woukJ be nice (perhaps) if all ^tudents In a- school were^interested In learjiing all they possibly could 

vyhether or not a check w«re made on their progress. This, fjowever. is rrot the^ case? A f ew will put forth their ' 
> best eHorts whether or not tests are'giveri. Biit the majority will work tiarder^f they know that they are to be 
held accountable for what h^been taught Generally, the instructor who admir)lstf » the mosx rigid programs y 
^ of evaluation gets the greatest arTX)unt of work out of his students. V ■ * i 

There is orie danger in using tests and t«t results as an incentive to students to *^ply themselves .to'^work and 
, study. Theirjinterest in marks can be a Superficial onp>which easily leads to dftorty-to "hit the test" rather < 
. than learn the subject matter for its value now and ia the future. Students who study primarily to pass tests 
usually forget the material n.jch fas^er'than those who are iriterested 4n learning because of the values to be « 
derived. A positive suggestion is this: Give rigid t^ts; give them frequently; but design tests that require your 
studen^to rhake application df vyfiat has been taught*lMicheels 8t Karnes, 1950^. 89). ' * ' 

Students tend to study tho^ things most likely^to appear on examinations. Therefore, it follow^.that 
students in electronic courses; usually studv; theory and job knowledge, rather than practice job 
performance. AH such ccfUrses use ^aper and pencil tests of theory and job knowledge. It is safe to assume 
that at jeast some electronic technology "courses canfce found in which no formal examinations or tests are 
given concerning (the students' ability to perform job tasks, . - ^ 

\r\ other courses each student may perform the same tasks In hi^ Tabbratory or shop practice, tasks, 
which shpbid appear in job task performance tests. |n lieu, of a perforn^ance test score, he wlj begivena 
grade or 'score on thfe work of h|s laboratory manul*. Ordinarily, the woVk done.in the laUoratS-y i&of the 
"typical" effort type rather than the "maximum" type (Lyrpari, 197.1). The student isnotfaced with the 
problem of having to d«monstriate his ability to perform these tasks in a formal examination. Under these 
conditions, he' will not '.spend as large an amount bt time ^vigorously practicing jqb behaviors as he will 
practicing the verbal .'bittaviors required by the fojmal written examinations. The student also makes a 
special effort during the/aTPT. The writer has observed students practicing such task behaviors on, their own 
time in preparation for such performance tests^T^eachers also tend to ptece more emphasis on. teaching 
fprmally tosts behaviors- than they do on teaching those behaviors evaluated in sbme^less demanding 
manner. ' . 



Figure L Effect of test^ on leamii:^ effort. 



Wijliams Whitmore (l959) reported a study in which a verbal theory ^examination ai\d*a job 
sample performance t§it were ^ven to graduates of an Army electronic maintenance couije. A written 
theory and jbb knou^fiElge test was developed to measure the retention of the knojyledge acquired by 
technicians during theB school training in electronics. The performance jest w,as made up of actual job 
problems using 'actual field ^uipment. ^Both . tests were administered td ^91 -graduates at the time they 
• completed school, and both tests were administeredjo another group of 98 graduates with from 1 to 57 
months of field experience (mean 19 months) who had previousljj^completed the same training. Among the 
resulting conclusions were the following^ • 

' a^Tli??fcW<xperienced ^oup sf^ed higher on the performance tesi but jower on the^rittentest than did ( 
the inexperienced group, with a trend p6wM higher performance and lower writteii scores as the atnount^f 
field experience increased, jind b, the contrasting resuHs in the written tests scores foDowing ^aduation raise . 
a question as to whether some of the material in the NIKE AJAa IFC Maintcjutnce CojursQ is rdctant (o the 
^ » field job (WiOian\s& Whitmore, 1959, piliO: . ' ' ' <^ • 

The written examination usdd by WiUiams aftd Whitmore was based orfthe content of the United 
States Army NIKE course and^the performance test used sampled the tfctual jo^ behaviors. The correlation 
between the results of paper and pencil theory tes*t and JTPT are.veryjow, Wlliamsand Whitihore found 
only a .144 correlation between* basic electronic (theory) pa{)er and pencil testicores arid JTPT scores for 
inexperienced graduates. For experienced graduates, the correlation was .196. 4^student*s performance on 
tests of knowledge of theory seems to iA\ very little aboutihis al^lity to perfomi job tasks. Also, it does not 
predict his ability to learn such job talks very wdl. The^first of flv^ recommendations of this study was 
that: • ' * - y 4 - • . ^ 

The Air Dcfcree^School review the training objectives fy tfie NIKE AJiSc^ I.FG ^technicians course, and y 
analyze current course content to determine its job relevance, and some fori^ o{ the performance tests 
X • developed in thb study be used to evaluate the effects of any changes made in NDCE' AJAX course 
\ (Williams & Whitmore, 1 959, p: iii). *. , / i 

Several other studies, ftich as Brown et, al., (1959), and Shriver(19^d)ripaSt Jurther light on the lack 
of job relevance in much of the theory content of electronic jtechnplogy courses, Anderson (K962b) made a 
. study of the amount of mathematics used by Navy electronic tephniciaris'jbillfound that very little vv^as. 
actually used on the job.* The mathematical contenjtof thes^ courses tnakes th^m very difficult fox many 
students. * ,» ' ' * \ ^ >' ' 

Examples of lacic gf Relevance in many curriculum items of the "liS^fonal theory,' courleS can be 
demonstrated by careful analyses of thesfejteins. A casK^al, uncritjcaJ examination of the same items could 
result in the conclusion that they are rdevant. Th^s^Jbjasj of the \^acuum"tu^e vdltmeter'ls prese^nted as'an 
example. On the job, the technician is required to 'use this meter daily, and lie should be able to use it 
precisely. In a job oriented training program, the studesnt learns* to use' a vacuum.tube* voltmeter by 
practicing its variqus uses. He leaiois to hook up the meter tq the proper points in (|ie equipment being \ 
tested, to select, the propw voltage range' pf'the met^s switch, and to read voltage values accurately .on the 
proper scales of the meter display. In this situation, he pays lijttle attefltion to ".the internal theory of 
operation of the vacuum tube voltmeter, much as the driver of an autonioJS^ile normally pays little attention . 
to how the engine in his automobile ^rks while he is driving.' . • *, / 

In a theory^ course, under the^same heading of vacuum tube voltmeter, the^Mudent learns a different 
set of behaviors - the ability to verbalize and to analyze hoW thexircuits of tjie meter a/e designed using a 
symbolic representatioft^or schematic. It is possible for him to Jearn^^uch behaviors and never actually use 
or see a voltmeter. Hie latter behaviors are seldom, if ever, required of tli«echnicteaon hisoob. Althou^ 
such ve,rbal knowledge probably do^s the student no harm) it certaiWy confflbutes little to his ability to use 
the vacuum tube vdtmeter in job situations.* Foley (1969) reports fliat Air Force technicians^ who had been ^ 
upgraded on the basis of a job knowledge t^ts, dould not u^ lest equipmei\t accurately when given a JTPT. 
Anderson (1962a) reports a similar situation in the Na\|y. " j * * * ' ' , . > 

Flnucane (1966) reports an Army situation oft an apparent spare part^s'hoKage for*t|ie repair of 
engines. An investigation indioated that repairmen were discarding«gopd' parts* as* being faulty. The 
investigation also indicated by use of JTPT that the repairmen qould not locate and i5Jentify^defective,i)arts 
even though they had been upgraded on the^baSis of a job knowledge tfst. Hljdtt and Joycii(1968) and 
Shriver, Rnk, and Trexler (1964) have demonstrated that electronic job tasks can be performed By subjects 
wfth little or no knowledge of traditional theory or principles. Such ^{ibjects, however, must be proficient 
in the use of test equipment, handtool?, and wdl designed job performance aids. / /• 




The job*oriented studies summarized in Foley (1967b) indicated that students graduating fro^ri job 
•oriented courses, in which they recjeived considerable practice in performing job tasks, were able to perform 
the assigned job tasks ahnost immediately. Students of simUar aptitude, graduating from theory oriented 
courses were not able^ to perform the §ame job tasks untU after^five or six months of experience 6n the job* 
The results of a number of independent critical examinations of the theory content of various Armed 
Forces electronic technician courses, in light of actual job content, are the basQs for concluding that 
ttaditional theory content has doubtful relevance to the. tasks of jobs for which personnel are being 
prepared. { 

In vieW).of existing evidence, the yalidity of the contention that a knowledge of electronic theory is a 
necessary prerequisite for th^ performance of electronip maintenance tasks is suspect. Even though this 
contention^ is deeply imbedded iHi the culture of the electronic maintenance community (Figure 2), it 
certainly 'shoul^ be subjected to a large-scale, objective, controlled evaluation to determine its empirical 
validity.. However, as long as the learning of electronic theory Is a stated objective of electronic maintenance 
courses, students should be tested as to how well they learn this subject matter. The appropriate^* 
instruments for such measurement are paper and pencil tests. But the results obtained from such paper and 
pencil measurements should be reported separately from thi results obtained from criterion referenced 
JTPT.. • . " . . 



, , , • ^ IX. GRADES, MARKS, AND TEST SCORES 

* The Aih^tan scHool tradition requires some form of a grading system. A search of the literatureyn 
the Education Index reveals that a great many articles are produced each year on this subject. No r tfyigfl tes 
were found in the same indek regarding the criterion problem, the answers to which are basic to good 
testing and grading procedures. Stanley (1964) says that "changing scores into grades is at best^'a rathep 
jftrbitrary process, and this arbitrariness is further complicated by the public-relations aspects of reporting to 
■parents." Grades, however, cjan be no better than the test procedures that^ produce them and because of 
their limitations, probaBly not as good. Even if the tests used in a course^do tefledt the objectives of the 
course, grades will not be completely satisfactory as TraVers (1950) indicate$J "AnJ' system of assigning 
grades inevitably includes many unsatisfactory elements since the scale does not, and p,ossibly cannot, meet 
the criteria bf a satisfactory measuring device.*' 

\ Stanley (1964) indicates that most modern report cards contain both grades and checklist items. 
Scholastic achievement is usually measured by ajetter system such" as **A, B, C, D, F,'* while citfeenship is 
in'dicated by an adjective system such as S (Superior), N (Normal), and U (Unsatisfactory). He indicates 
that achievement, as measured by tests, should not be combined with characteristics evaluated subjectively 
such as "effort,** punctuality, deportment, and neatness of work. If the results of testing are to be reported 
ac(Jurately in the grading system used for electronic training courses, the achievement graces should not 
reflect effort. Such a'practice decreases th6 validity of the grades as indicators oPa student's proficiency in 
the job task performance objectives of the|course • . 

As previously discussed, criterion ^efjftrenced JTPT (used to^ measure ability to perform electronic 
maintenance job tasks) and norm referencedipaper-and-pentil tests (used to measure electronic theory and 
job knowledge) are to a great extent measuring , different factors and are based on different measurement 
technology. Grades derived from each'^source should be reported separately. This would permit the users of 
grades to make separate judgements about these two aspects of^the training progranVas indicated by 
Frederiksen. , ^^g^^^'^'^ ^. • - * ^ * - 

Criterion measures which more accurately reflect the objective of instruction and which permit judgement^ 
' to be made separately about various aspects of the teaching program are needed. / ^ 

In recent ycaxf there seems to have been a rapidly growing awareness of the need for training evaluation sind 
*fo? developing proficiency measures for use in evaluation which better reflects the sought-for outcomes of 
instruction (I*rederiksen,. 1962; p. 323). - , , • * 

Unless separate grades are reported, it will be impo'ssiblelo determine the effectiveness of the theory ; 
aspects and the job aspects for electronic technology education in job success follow-up- studies. Separate 
grades also can be justified on the bases of better communications. Travers^(1950)Jifdicates trfat: 



Due to traditional influences the elimination of traditional theory is probably impossible at thi$ time. 
Perhaps substantial reductions could be made gradually. Some of the real reasons for jhe persistent 
demands for iheory should be examined. In spite of the experimfental evidence to the contrary, many 
electronics teachers and many employers of graduates , demand- a knowledge of traditional electronic 
fundamentals. Pickering and Anderson fpund this attitude.among supervisor^in a 1966 follqjw-up study of 
graduates who had completed an itxperimental performance-oriented electronic-technology course in the 
Navy. y,', . , . " ' ;^ . ' 

Prellmijiary follow-up results Indicate that the experimental ET (electronic technicians) ^raduatei have been 
- gen^lly well received In the fleet However. Ifis also clear that ^ subtte but widespread bias agairwt them 
exists* For example, several super^][sor5 have made comments simiJaf to this: "X Is a good man. Hq*s done 
everythlng'Vve asked him to do. 6f course. \ vroukJn't assign him to a reaily difficult job because he hasJi't 
had enough theory/' Judgment? of limited capabilities are being made Independent orf objective evidence. 
Regular ET A school graduates have succfssfully completed acomptex and difficult training program that has 
placed emphasis on the attainment of knowledge atiput theoretical concepts and demonstration of capacity^ 
to manipulate electronic relationships mathematically. BVt as a group, are justly proud of their - jf^ 
accomplishments. The experimental ET school graduates are not considered part of this group. What effect-:;;^ . 
I these attitudes will have on the careers of the experimental ETs Is not yet evident, but the attfiudes ar^ real 
enough (Pickering & Anderson. 1966. ^.39). * . . / 

If the supervisor is^ an electronic technician, he has successfully confipleted a rather lengthy and 
difficult electronic theory or fundamentals course and hp now is able to perforrp the job tasks of an 
.'electronic technician, at lealst sufficiently well to "get by." Evet^^though he probably learned to perform 
motst.dt these tasks on the job over a period 6f several years, he may firlnly believe that completing the 
dif f iculJcJh,^orY*course is the key, to his success as an electronic technician. A new man is not one of the "in 
group'^'iftlless he has been initiated)n the same way. The new nrian will not be pernrytted jo demonstrate his 
ability' to perforrft until .he demonstrates that he can give "lip service" to the theo^* The writer has 
experienced this same unquestioned belief in ''the value of theory iahis contacts with^aiige numbers of Air 
Force, Army, and Navy school and field personnel. He^also has found the sarrie bias^mong nribst of the 
civilian electronic t^nology course instcuctors he has contacted, Jt^e recjuirement that technicians have 
s^h acknowledge of theory has been further reinforced the C5epartment of pe^en^ by the 
theory-oriented job knovyledge tests that each person must paS^'teifore he is promoted from apprentice to 
specialist to technician^ . ^ ' ' 

Figure! Why tliory trainipg? , . ^ ^ 



A' grade system is a measuting scale, but before it can be u$e<) as such» it is necessary to know what it is 
^ mcasyring. At tHfc present time, student grades are used to measure^it least three different things. The grades 
given 1)y one tcather jmay 'indicate the extent to which the goals of the course have been achieved in iht 
^students. The^grades given by a second teacher may indicate the relative amount of progress made by a 
* student. The grades given by a third may indicate the relative standing of the students (gijding on the curve). 
According to the system of the second teacher, a student who starts a course with a poor background and 
ffinislies with average proficiency hiay;deserve a 'better grade than one who starts out with a good bad^roura 
and finishes ^thc course with a high level of achievement. A schooUshould determine what is to be measured 
by grades so that those who use the cumulative records may know how to interpret them. Most cumulate ' v 
records cannot be interpreted because the grades of different teachers mean different things (Travers, 1950, 
p.145). , ^ ^ • ^- - ^ 

^ Scores for electronic job task^ performance tests have lon^ been a problem especially where 
discrimination is desifed..in course achievement tests; "go/no-go" standard of performance within a time 
limit can be applied, as^^rted by Williams and Whitmore (1959). they obtained a differential score by 
converting time required to successfully complete the task into a standard score. Hansen (1958) who has 
studied the relative merits of electronic troubleshooting job-task performance test scores, has recommended 
that several scores be used: . » - * . 

1. "Unie required to complete the checklist activities in symp(bm««eeking phase of the test.** The 
writer has classified ihis type of behavior as checkout procedures. . . ^ 

2. *Ttme required to complete the trouble4ocation phase of activity." this is similar to the WUiams 
and Whitmore (1959) procedure. " ' ^ " 

^ - ■ . 

3. "Number of discrete steps of trouble-locating activity." ^ 

4. "Numberoferroneous replacement units." (Hansen, 19S8, pp. 4i -42). ^ - 

Non^ of these suggestions is truly satisfactory. Ute whole problem of.how to score and rkport the 
results of JTPTis certainly a worthy area of exploratory ^velopment pne promising solution ,to the 
discrimination problem has been suggested by Dr. Gordon Eckstrand of th^ Advanced Systems Division of 
the Air Force Human Resources Laboratory^ i.e., the training time that is 'required by ^ student to reach a 
"go,^o-go*' standard of performance fot each task tested. A variation of this approach would be the 
number of testing attehipts required to reach criterion on each JTPT. In the meantime, ^^tever scorcs^are 
used to determine the iheory grades an^ job task performance grades, and the meaning <^fth^e two types 
of grades biased on different scoring systems should be adequately communicated to t^ users of these 
grades. In order to-further jmcrease the meaningfulness of the job task performance gradef me writer would 
recommend that it be supplemente4 by a list of activities included in the course job task performance 
measurement ^stem, together wUlran adjective rating of each student's quality of peiTpr^nce for each 
activity. Such a pjoflle of a student's job task pyroflciency would also emphasize the nnportance of this 
course objective to the students, instructor, schoo^administrators, and maintenance supervisors. 

X SUMMARY AND CONCLUSIONS . 

L This report* represents an in-depth analysis of the state-of*the*art concerning measurement procedures 
for as(;ertaining the job proficiency of electronic maintenancepersonnel. It is based primarily on a review of 
the literature made as part of the writer's doctoral, dissertation (1967a). It also, reflects several other efforts 
and" events supported by the Advanced Systems Division, Air Force Human Resources Laboratory, 
Wright-Patterson Air Force Base^, Ohio,anclading a bibliSgraphy on the measurement of niaintenance 
personnel j)repared by Dr. Askren (1963), a paper on the status of technology of training by Dr. Eckstrand 
(1964), a field survey of maintenance practice (Foljey & Elliott, 1967)i a 1966 conference on the 
Assessment of Complexed Operational Titsfa arranged and reported by Dr. Chiles (1967), imd a 1968 
conference on human factors testing arranged and chaired'by Mr. Snyder (Snyder et al., 1969). t 

2. The first consideration that should be addressed when a test is ^eihg developed is what is It supposed to^ 
measure. Whatever it is supposed to be measuring is its criterion. In the^cas^of tests concerning the ability 
to perform electronic* maintenance, the* ultimate criterion, therefore, is tjie tasks of electronic maintensince 
performed in their job environment. For testing purposes it is usually impossible to slniulatc all of the 
conditions of such an ultimate criterion and at the same standardize test conditions. But it is pqssible to 
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require the test subject to demoraUrate that he can ^rform key tasks of this job removed from their actual 
job environment. A performance test battery, made up of such tasks is considered to be .very close to the " 
uhimate criterion. The difference between the re^al world aijd the simulation of the real world for'testing 
purposes ircalled th9 cri)erion problem. Many times paper and pencil job knowledge tests or supervisors' 
ratings ^re siibstituted for )iear ultimate criterion tests, making the criterion problem greater., TTie farther 
removed from the critetion that testing procedures become, the^more the danger that the tests are not truly 
measuring whal'they are supposed to be measuring. The degree to which tests relate to Ihe criterion is 
called the degree of empirical validity. ' ' . ' ^ 

3. 'This report indicates that. two very serious criterion problems exist in many electronic training 
programs^ The empirical validity pfther paper and pencil tests used to measure student achievement'has not 
been determined and the job relevance of much of the trWing content has not been,ascettained. 

4. The full application cff mc\dern training development tarfmology based on a^ystems approach supplies a 
realistic sokition to both the problems-. A key element of the^echnology is the care used in deriving training . 
objectives based on a sy/tem^c identification of the tasks of the job or jobs for which the training program 
^ being developed, foIlowecHiV a systematiq analysis pf each identified task. A second key element ib 4s 
important ^s the first. i\t4he» time the objectives are bein&Hjrmulated th^ criterion measures are ako being 
developed. Based on the analysis of each identified task, the content of the job oriented training program is 
developed. The criterion measures are based on the objectives of the training program,jiot on the contend 
of the training program. The criterion measures, appropriate for determining student achievement^ in 
electronic maintenance courses developed in this manner, are Job Task Performance Jests(JTPT). 

5. The development of the core content and' structure of current electronic maintenance training programs 
predated by many years !he modern course development technology. The core contend of these courses 
reflect the influence of the electronic design engineering community. The courses are theory oriented and 
^afiedi the subject matter structure found in most .electronic engineering programs. The tests used to 
determiner jtiident achievement are based on the content of this academic subject matter and arexrf a paper 
and pencfl variety, which of coprse are appropriate for measuring academic achievement. 

6. In addition to this more or less standard theory content, current electronic maintenance training also 
contains instruction on typical electronic equipments or sets. SUch instruction may be theory oriented or 
job oriented or both. But the job orientation of such training programs has not b^en achieved in fhef careful 
manner recommended by. the modem technology for training development. Some JTPT are administered 
but the main thrust of measurement is of the paper and pencfl job knowledge type. The scores obtained 
from all theory, job knowledge, and performance tests are combined to produce a single score indicating 
school success. This is an inappropriate^action. Most paper and pencil tests measure student success in; 
learning theoiy and job knowledge, and are norm-referenced. JTPT measure ability to perform key job 
tasks and*are criterion-referenced. Mixing the two types of scopes gives a meaningless result. * . / • 

7. The graduate of such training programs is assigned as an apprentice to a field maintenance unit where ht 
^ receives on-the^ob-training Which is certainly job oriented. However, he seldom if ever is given formal JTPT 
'to determine how well he can perform the key tasks of his job assignment. His advancement to tbe 

journeyman level is greatly. dependent on Hs ability to pass a paper and pencil Specialty Knowledge Test 

which is of the job knowledge test variety. 

* . •» ' 

8. The above analysis of current electronic training and testing practice indicates that great ""reliance is 
placed on paper and pencil theory and job knowledge tests for determining training and job success almost 
to the exclusion of criterion JTPT. This volume identifies. Consolidates and reviews.the data fobndin the 
available reported research concerning' the empirical validity of paper and pencil theory and job knowledge 
tests and school marks. The correlations, obtained when the results of most of these tests have been 
compared with the results of applicable JTPT, are very low. TTiis would indicate that the empirical validity 
for most of these paper and pencil instruments as well as. school marks qbtaineflfrom com})ining their 
results is suspect. Tlie limited kmount of rescarch^literature found concerning peer ratings andf supervisor's 
ratings also would indicate that their empirical validity is suspect. " 

9. Several paper and pencil t^ts concerning troubleshooting such as the Tab T^t and the - 
Multiple-Aliernative Symbolfc Trouble (MAST) Test are reported in the literature. It would sdem'that such 
tests would test the cognitive factors of troubleshooting. However, the comparison of the results of these 
tests and the results obtained from JT?T reflecting the same equipment troubles produced rather low 
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correlations. In the opinion of tHe wnter,.the JTPT*contained many distractions; normally present on the 
job, that t^e tests based on the pure cognitive processes did not contaii\. Considering this, the empirical 
validity of such paper an4 pencil symbolic substitutes could possibly be improved by further development • 
and refinement. In view of tlie ^pensK^e equipment requirements for most JTPT, the development of 
symboUc.substitutesofhi^ empincalvalldity wpuldbe very deMrable. ^ ^ ' * ^^^^ ^ • 

10. There areJj^gnv studies concerning tAe relative effectiveness of theory oriented and job oriented 
electronic maintenance training reported in the literature. The results of these studies indicate that the ^ 

" sraUuates^of th$ job oriented training programs are able to perform pr.bduclive work immediately upon 
assignment to field maintenance units. Ihe criterion' of school success in these courses is the abUUy to pass 
JtPTi rathex^an {)aper aiui pencil tests .on theory and job knowledge. The percent of students of hi^ * , 
electronic aptitude who successfully complete either theory orv^nted of job oriented training is about the , 
same. A much hi^er percent of students of medium (not tow) aptitude successfully complete job oriented 
" * training. In some oases almo^ no medium aptitude students are able toisuccessfuHy complete tiie theory 

. portion of the theory oriented.programs. * | 

11. Such result? would * indicate that much of the theoryt content of ti:aditional electronic training > 
programs^ is not relevant to the performance of the maintenance tasks performed by personnel in their first 
enlistments: However, the contention that alcnowjedge of theory is a necessary prerequisite to the 
successful performance of maintenance is deeply imbedded in the culture of the electronic maintenance 
community. But there is no' doubt that this prerequisite adds greatly to |he personnel costs of the Air 

iforce:. , . • , • . ^ f ' * • \ 

12. - This report has identified many problems and questions conc^eming.Air Force maintenance. These i 
problems and questions concern the adec^uacy and efjficiei\cy of the pen^nnel selection procedures, the - ' 
personnel training procedurips, and field maintenance practices Since most of the reported research has ^ 
concernedelectrorac mainteioance^ this reporf hias'been concerm|d primarily with this type of maintenance. . ■ ^ 
Bowev^f^^ik^hof this research wodd also apply to mecharUcaljiAain ' \ 

13* The findings and anafyses of this volume indicate an urgent jrequirement for ^well planned, adequatdy ^-i., . 
^ funded, comprehensive e;q>lo*ratory and advanced develc^nient nrogranu for syftematicaOy addressing and. 
- solving these problems. The first effort of such a program would be a comprehensive, an^ in-depth st^dy, * 
* based on hard data, concerning how wdl maintenance men can perform the kej^'tadcs of theirjbbsi Sueh,an 



^-depth study would t)rovide a needed^base and feedback for foQow oti ^die|:for the improveinents of 
^^'^laiotenahce prafcticejpersoim '' \ 

. \4^> \A s6rjiOM^ gap^ln the modeni technology of training development, is.the lack of adequate guidance 
.how to dev^op arfiT'administcr criji^rion referenced ftPT. Prior to the work on criterion reiTercnced ^Tl , 
rep<Jhcd to. Volume fl of this repprt, ypry little exploratory^ development on JTPT tecl^Iogy had been | 
iu^pojied since the late l^SOs. There if an urgent requirement for a modern^ guide 'on howJto develop JTPT / 
for mffltarytechnical training . 

' . XL ACTIONS AND RECOMMENDATIONS . . . 




Acttons 



Based oh-ineianalyseJ ah^ conclusions presented in this volume, several actions^have already. beerr . 
taken. * ' ^ • . • . % \ ' i^"* «sV</^.vrs^ 

. l^A* contractural effort '.residted' in^Jie development of. Job Task Performance ffestS:,(Jl^ arti 
appropriatb scoring schemes fof tach of the key electronic makitenance job activitie^deiitifie^^^ 
volume (Contract 1'33^5*69-C4232), The work statement for tjiis contract is sup|pBed as^^p^ndbc A;/^ 

2." A folow-on contractual effort resulted in the development of a detailed Test AdrpimsWon Mariual» a 
training program^for Te^t ^dhiinitojors an4a.limited tryput of the JTPT (GpntSct:J^e61$46-fc:i^^^^ 
The results of both these JTPT contractual efforts are repoited in AFHRI^TR«74*57 (lO»0trtstan^^^ 




3. Two other fotfow-on contractual efforts were initiated, which rpsuhed in tke development and limited 1 
validation of both graphic and video symbolic substitutes foreach'type of job activity forwhich JTPThad 
b^en developed (Contracts F33615-70-C-1550 and F33615-71-C-i505). The results of these efforts are 
reported in AFHRL-TR.74'57 (Volumes HI and IV). • . ; 

r • 0 ' 

Recomroeiidations ^ . ' » 

>• ■ . , , ^ ' . ■ 

1. Comprehensive exploratory and advanced development programs shouTd.be established and adequately 
funded that systematically identify and solve problems* concerning maintenance practice in the field, and 
problems concerning the selection and training of maintenance personnel. ^ 

2- A ii^essary first effort of such programs should be a comprehensive and in-depthstudy, based on hard 
data*, concerning how well maintenance men can perform the key tasks of their jobs. Such an in^iepth 
"study ^wouldjequire the development and use of JTPT for electronic and mechanical maintenance jobs. The 
results would produce criteria for determining the validity of personnel selection and training prograir^. 

3. A rnoderfTguidB on how to devfelop JTPT for maintenance jobs should be developed. Such a -guide * ^ 
should'^bc dn xm-^g'^ffort. This^^ide should be developed now, based on our current *|cnow how.^* The^ ' 
^ide should be ijf^^d periodically as more expertise and sophistication in the devdophient and use of v V 

such tests^re achievftS. • !' ' i ^ \ > - . 

4. In tr^ning pjograms in which both paper and pencil tests and JT|T are administered, their results ^ \ > ' 
should be reported separately. The results of JIPT should be reported in the form of a profile rather/th^an V' Vv:. 
under the umbrdia of a single meaningless score. ^ * / * , ' > V '> ^> 
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APPENDIX A: WORK STATEMENT ^OR CONTRACT . 
F33615-69C-1232 CONCERNING DEVELOPMENT OF CRITERION ^REFERENCED 



JOB TASK PERFORMANCE TESTS 



. This work statement refl^ects ^findings and recoiomeniiatlons^of 
AFHia-TR-74-57(I) as well as .othef-Vtforts mid events of the Advanced 
Systdms Division of the. Air Porcre^ Human Resources Laboratory indicated 
In that volume. This 8 November .1968 version of the :>70i:k 'sta|:6ment was 
included in the contract • However^ the f irst ,dr4ft -was 'CompJ||Bted In 
June 1968*' It Is included here /to lldlca^^e the infopnation :and 
directions made available to 'the confracl^or at the time; he '^tar ted his . 
exploratory development effort* * \v ; ' ' 




Date: 68 Nov 08 . 

DEVELOPMEOT OF JOB -TASK .^PERFORMANCE MEASUREMENT ' 
r ' ' I I'ROCEDURES FOR ELECTRONICS -MAINTEl^ANCE AipiyiTIES 

1 ,0 INTRODUCTION (Objective) :. The primary purpose of* the effort. Is 
to devflbp adequate i>rocedures^ for meiasuring the ablllfc'y of subjects 
\ to perform each type \of job activity described and to develop an 
\ appropriate scoring scheme for use with each .measure. It Is antici- 
pated that these measuring procedures and scoring schemes will have 
general applicability for measuring an individual's ability to perform 
,| jDoaintenance activities for a part;icular weapon system or equipment.- 
The^e job-task performance tests j^ll be used In later exploratory * 
development as criteria for future development of symbolic tests. 

2,0 SCOPES The yehffele for developing these measurement t^rocedures 
will be the Radar* Set AN/APN 147,^ the accompanying cqmputer set AN/ASN 
35, and their associated test equipment. Job task perfd.rtnance tests 
and appropriate* scoring scheme^ shall be developed for j'oi> ta^sks 
activities associated with this, electronic system, for information 
gathering activities (using test^ equipment) , and for handto^l acti- 
vities. The test procedures used for meastirlng the job -activities 
, asspclated with the' electronic equipment should^be .applicable^ to any' 
r^lectronic system but the tests will be developed for the AN/APN 147- 
AM/ASII 35 system. The performance tests concerning the;use*of each 
, ' test equipment will include all the capabilities 'qf f6e<^test equipment; 
The handtodl tests will be^llmlted to the haxid tool, activities assod-- . 
ated^wlth the AN/APN 147-AN/ASN 35 system. * , 

.3*0' GjaqERAL BACKGROUND: liii^'preseht day electronics technical train-; 
Ing' programs, many formal paper £md pencil tests are utilized for / \ 
, measuring and predicting student achievement. Research In ascer£alning 
the effectiveness of such paper, and pencil predictive and achievement 
tests, in ascertaining the effectiveness of various training methods, 
' and in determining on-the-job effectiveness of. electronics repairmen 
-:-^'^d technicians ^has been hampered. >y a lack of job-task performance 
; f measurement and scoring procedurej&"for various job activities.. Recent 
' study of these activities has resulted- in a plan for developing X 
effective measurement procedures. ' Rather , than deve|.oplng one' test and 
one scoring procedure for the entire'^range of job activities, the 
>^ ^development of an appropriate^ tfesting-procedure and scoring ^scheme for^ 
, each^df several classes of job ac tittles of electronic /te^chnl^cians is^ 
>4>roposed. This, exploratory developm^t effort is bti|t: the^first step. 
qt a plan for Imj^roving^tfte effectlv«ess of Air Force measurement.. 
I The successful development of valid a;cid workable proicedures will ,j?r<^vide 
\ th^ informatioii fot gfeatly improving* the effectiveness of training and 
; I 'of job performance aids* 'fhey^.wlll provide a criteria for ascertaining 
the effectiveness of cutrent predictive and achievement measurement ... 
practices 1, Th§y/wilL alsoxbe used as critjari^ for future explprat;pipy * 
— -development of valid paper and pencil substitutes.' 



3.1- Applicable Documents List: A comprehensive search has been 
made of the measurement literature Including a DDC bibliography seiarch. 
The most applicable technical reports, books » and articles are listed* 



3.1.1^ Technical Reports > These reports Sre'^bn file at the 
; Defense Doctanefitatlon Center (DDC), Cameron Station, Alexandria, 

Virginia 22314. Each, report's DDC number is included in its descrlp- . 

tlon below; I.e., AD ' They can also be obtained from the 

Clearinghouse for Federal Scientific and Technical Information (CFSTI) , 
Sills Building, 5285 Port^ Royal iJoad, Springfield, Virginia 22151. 
(Since all of the references which appeared here in this work statement 
. are included in' the references of this volume,* only the ajuthor and^ year 

of publication are given here and include: Anderson (1962ib), Brown et 
.^al. (1959), Crowder et ,al. (1954)., Grings et ali (1^53) ,^ Harris and 
^ Mackle .(196i), Mackle et al. (1953, and William and Whltmore (1959):i. 

In addition, the contractor watf given a copy of Foley (1967a).) 

3.1.2 Bocgs. (Frederiksen (1962X, Super and Ctltes (1962)^ 
Thorndike (1949) and Wilson (1962).) > 

* V3.1.3 Articles . (Jenkins, '1946; Wlace, 1965) 

4.P TECHNICAL REQUIREMENTS/TASKS:" • 

4.1 Areas of Conslderatlort^ ^ It is emphasized that the primary 
objective of this effort Is the development of^ adequate measuring^ 

X procedures and appropriate scoring schemes t6x the various job activi- 

ties of electronic technicians. , These procedures and scoring scheme^ 
1^ ^ should have general applicablliey ^for measuring' job effectiveness ip * 
\ any electronic maintenance career field. An .Electronic maintenance 

technician performs maintenance tasks on an Electronics systeijd or group 

* of systems making use of the 4:apabiiltrles. of electronic t6st equipment 
» to obtain Information about his systems.^ "^j^^!^ maintenance tisks 

include the use of appropriate h^ndtodls. This^ effort .is laimed lit the 
diagnostic measure of all the activities included in the maintenance ^ 
process. • it * ' ^ 

4.2 Tasks to be Performed ?^ f ; 

... , ' ' ^ ""W*- r ^ ^ ^ ^ ' , ' . * 

. *■ . 4.2;1 The Disvelopment of ^Job^Tasfe Performance Testis for * 

t Job Activities Associated* with Electronic Equipment . The contractors 
' ^ shall develop measurement procedures and appropriate^ scoring, schemes 
.} for each of the following activities associated with the/ maintenance 
of electronic equipment: <!,) performing equipment checkout, procedures; 
* - (2) adjusting and aligning; (3)|isoiatlng lietween-stage faulta 

• particular state (or functional unit or physically replaceab^^^ 
(4) Isolating vithinr stage faults to defectiye co^pongntsy ,subh\'as 
tubes, splid state devices » 99^1 > capacitors or- iredist^^^ 

, /replacing of defective compoi^ents. The^xontractpr shali 
^ requited to state his ratibnafe for seMcttbn of ihe patWcUlar^^ 

t TRir ■ - ^ - ' ' '^^'^ - ' ' ^ ' • Q^'"^ ' h , : - 1^- - ' 



measurement procedure and scoring sclfeme for each type of activities. % 
These procedures will be demonstrate^ by developing performance te^t 
appropriate scoring scheme for measuring the adequacy of each, of the 
following AN/APN 147.-AN/ASN 35 maintenance activities:* 

4.2.1.1 The entire^ checkout procedure for 'the 
M/APN 14'7-AN/ASJl 35 system (1 test). . 



4.2.1.2 All of the alignment procedures for this 
system (;l te^tj . . , ' ^ 

i 

4.2.1.3 Selected typical adjustments foundifin the 
system (5 tests) . All adjustments should be identified and five (5) 
should be selected. These should be adjustments other than the routine 
**front panel" adjusttnents" made during the checkout procedure. 

4*2.1.4 Localization of be tween-st age .faults to a 
particular stage (or .functional unit or "physically rep ],aceable unit) (1 
test with variable troubles). Thls> testing procedure should ^he such 
that any defective' unit could be Inserted lii the system and be " 
identified as defective. - ) 

^ 4*.2.1.5 Isolatiotf of jd.thin-stage fault to defective 
-component (10 tests one for each of 10 selected stages). All stages 
or functional units shquld be identified and the 10 most typical stages 
or functional units should be selected. Each test should be don- * 
structed so that different defective components can be inserted in the 
stage. \ . [ \ 

* 4.2C1.6 Removal and replacement of defective com- 

ponents (10* tests). Jdentlfy all t^es of component removal and 
replacement actions for the AN/APN 147-ANyASN 35 -system. Select ten 
(10) typical components removal and replacement operations for .this 
system and develop *a test for each of these operafltions. 

4.2.2 The Development of/ Job-Task Performance Tests for 
Information Gathering Activities Associated with Electronic Equipment 
Maintenance » The contractor shall develop measurement procedures and 
appropriate scoring schemes for each of the following Information 
gathering activities:^ (I) using oscilloscope to measure voltage and 
frequency, to compare waveshape standard and to make high accuracy 
time base measurements an^ comparisons; (2) using ohrameter to measure 
/direct- current resistance ^n electronic equipment; (3) using sigtrSl 
generator to Inject standard or known 'Ssfignals to equipment cf or 'test 
purposes; (4) using , tube checker to estimate quality of transi gators . 
The test equlp&ent normally used with the AN/APN 147-AN/ASN 35 *^stem 
will be used as vehicles for developing the performance tests. ''However, 
the performance test developed for each. test equipment will sample the 
full range of capabilities »of -each test Equipment and will hpt.^be 
limited to pnly those' capabilities required fbr maintenance of che 



.AN/APN 147-AN/ASN 35 system. Performance testp will be pirepared for 
the |ollowlng test equipment: / ^ . 



4, 2,2.1 Tektronix Mcxiel 545A'drc.^B Oscilloscope 

,472.2.2 Model 505 A - vacviujn tube voltdhmmeter ' 

One test for testing of measuring voltage 
(all scales^ . ' ' 

\ One test for mea8ur{.ng resistance (all 

scales) « . 

4.2.2^. Model WV 97'--'rcA ElectShlc Voltphnmi^ter 

One.te^t for measuring voltage (all ^ales) 

One test for measuring resistance Kail 
scales) .1 



^ 4.2.2.4^ Model SG-85/URM-25 RF Signal Generator \ 

4.2.2.5 Model TS 382D/U Audio, Oscillator 

4.2.2.6 Model CMA 544A Test Set % 
^.2.2.7 Model CMA 546A Test Set i ' 

4. 2.?. 8 Model TS- 148- Spectrum Analyzer ' 

^ 4.2.2.ii Model TV-2 Tube Checker 

4*2.2. TO Model 1890M-Transl8 tor Checker / 

4,2.3 The ifevelopmeSt of Job-Task Performance Testra for the^ 
Handtoo 1 Activities o£ Electronic Technicians * The • ult Ima te^^ <* J ec t lye 



of this Atr Force test" development program will be td have..4?8(;jing ' . 
procedures available for measurd^nfe mpjtt of the ^|^d topi Vq>erati<m^;^ ^^^^^ 
required for use.^with most electronic equipmentSi ^Thls effort fj^H^^ V 
^ ' however » be'^mlted to the handtool oper^atidns required ^ 

mainteSahce activities of the AN/APN 147-AN^ASN SSssystem^ othjl majority 
of the tiandtool operations -ate required to support .the llemdyi^ and 
replace activities. ^^The contractor wl^H be^requifed to/?accompllsh 
%he following: . \ ^ . ^ - y 



- r 4.2^.3.1 .iMtify each type of soiderini^eaak^ih;^^^ . ^'i^llS 

AN/APN 147-AN/ASN 35 aystOT ^^>^ ! < - v^; 

L 4t2.'3.2 Prepare a soldering exercise for tyipe '\[ " : 

of sold&rlng taskjand to develop |t scoring.schem^ refle.ctlng the relative^! 



^^^^ 
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quality of performance of each exercise. Consideration should be 
given to the use of the R-F Probe In the measuring scheme, (See ^ 
Reference para-. 3.1,1.7 above.) * 

4. 2 .3. 3 Using the' list of remove and replace tasks 
developed for 2^2.1.6 above, identify the handtools required for the 
mechanical portions of these activities A test and scoring scheme * \ 

should 6^ developed to reflect the Relative quality^ of performance for 
each of these handtools. 

5.0 REPORTS AND DATA TO BE DELIVERED: 

B001 - Progress Reports in accordance with Exhibit 19 J\m 1968. 
; B002 - Job -Task, Performance Measurement Procedures for Electronic 
Maintenance Activities in accordance with Exhibit B, 19 
Jun 68. » • i 

B0p3 - Technical Report in accordance ^^tl;i Exhibit B, 19 Jun 68. 

6.0 SPECIAL CONSIDERATIONSi.,,,,,^^ • 

/ - 

, 6.1^.. Requirements for Contract Personnel: The principal Investi- 
gator and the senior contract scientist must be mature individuals who 
had had' successful experience in the development of job performance 
tests» which were used in controlled experlment(s) and ^reported in published 
document (s)*/ Samples and descriptions of these 4:ests must also be available . 
for inspection. It will be necessary^ for these individuals, to work on militai|y 
suites in cooperation with operational personnel. It is, therefore, desirable 
that each of these individuals have had previous experience as electronic^ 
techniclahs in one of the, military services. A^ contract personnel working 

, on this research effort* wilX requlre SECRET clearances^ i 

^ ;^ ' * ^ . • ^ 

6.2 Air Force Support \ 



6.2.1 Government Furnished Property ; The contractor will be*, 
furnished the- following property as indicated within 30 days of the - 
effective date of ^ the* contract. ' * 

^ 6.2.1 J The AN/APN-r47 - AN/ASN-SS system. 

^ ^ Th^^syst^j^^ll be available for the ' ^ 

contra^or's use at Wright-Patterson Al^B^ 
Ohio. ' ^ * 



;6 . 2.1.2 Applicable electronic ti^t equipment. 



A 



This equipment will be available for tjie ; 
contractor's use at Wright-Patterson APB> 
Ohio. * 



\ 6.2.1.3 Copies of applicable maintenance technical 

* , orders. ♦ 

These^^wfrn^e furnished for use at con- 
tractor facilities. 

\ 

6.2.2 Air Force Personnel: The Air Force will furnish a 
maximum limit of three technicians as subjects for tryout of the per- 
formance test^s^ These subjects will be made available at an Air For 
installation. The Air Force contract technical monitor will make 
appropriate arrangements for this tryout at the appropriate time. 
The arrangements also will include sufficient appropriate space for 
the tryout. 
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